Skip to content

fix(cluster_setup): stop reverting artifacts.yaml on repeat installs,… - #134

Merged
kupratyu-splunk merged 14 commits into
ai-tier-gafrom
fix/k0s-install-image-upgrade
Jul 26, 2026
Merged

fix(cluster_setup): stop reverting artifacts.yaml on repeat installs,…#134
kupratyu-splunk merged 14 commits into
ai-tier-gafrom
fix/k0s-install-image-upgrade

Conversation

@kupratyu-splunk

Copy link
Copy Markdown
Collaborator

… warn on mutable image tags

configure_images() snapshotted artifacts.yaml/splunk-operator-cluster.yaml to a .original file on first run and restored from it on every subsequent run before re-injecting image values. Since .original was never refreshed, any legitimate change to those manifests (new operator release, new env var, new sidecar) was silently reverted on every re-install after the first. Drop the snapshot/restore step; the sed substitutions already only touch their own named fields, so rewriting in place is idempotent.

Also warn in validate_image_config() when a configured image has no tag or uses a mutable tag (:latest, :preview, :stable*, :dev, :nightly), since imagePullPolicy: IfNotPresent means re-running install without a new distinct tag will not actually upgrade the running image.

Description

Related Issues

  • Related to #

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test improvement
  • CI/CD improvement
  • Chore (dependency updates, etc.)

Changes Made

Testing Performed

  • Unit tests pass (make test)
  • Linting passes (make lint)
  • Integration tests pass (if applicable)
  • E2E tests pass (if applicable)
  • Manual testing performed

Test Environment

  • Kubernetes Version:
  • Cloud Provider:
  • Deployment Method:

Test Steps

Documentation

  • Updated inline code comments
  • Updated README.md (if adding features)
  • Updated API documentation
  • Updated deployment guides
  • Updated CHANGELOG.md
  • No documentation needed

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published
  • I have updated the Helm chart version (if applicable)
  • I have updated CRD schemas (if applicable)

Breaking Changes

Impact:

Migration Path:

Screenshots/Recordings

Additional Notes

Reviewer Notes

Please pay special attention to:


Commit Message Convention: This PR follows Conventional Commits

kupratyu-splunk and others added 7 commits July 16, 2026 22:36
Covers connecting an external Splunk Enterprise instance to the SAIA
backend — JWT signing key setup, issuer_uri fix, mixed-content options
(disable SSL workaround + generic TLS termination via LB/ingress),
AIPlatform CR patching to avoid operator revert, and correct Splunk
restart procedure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… runbook

- Step 2: verify port 8089 reachability from k0s cluster nodes, not laptop
- Step 3 Option B: fix saia_sok_url stanza to [saia_sok_configurations]
- Step 4: replace incorrect AIPlatform CR patch with direct ConfigMap edit;
  explain that splunkConfiguration.endpoint is the HEC endpoint (not the
  JWT issuer), and that the operator only fills missing ConfigMap keys so
  a direct SPLUNK_ISSUERS edit is safe and persistent

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…unk runbook

- fix(P3): broken Step 3 table anchor pointed at old Traefik heading;
  update to match actual Option B heading (TLS Termination via LB/Ingress)
- fix(P3): curl HTTPS check after disabling Splunk Web SSL expected
  "connection refused" but port 8000 stays open speaking HTTP — a TLS
  client gets a handshake/protocol error, not ECONNREFUSED; updated
  expected output to match TLS negotiation failure
- fix(P3): restart-as-owner instructions only covered the case where the
  SSH user IS the owner; added sudo -H -u <owner> form for the common
  case where admins log in as ec2-user but Splunk runs as 'splunk'
- fix(P2): Cleanup section still referenced an AIPlatform CR patch that
  was already corrected in Step 4; replaced with direct ConfigMap patch
  for the both-issuers case; also replaced stale AIService endpoint
  check with a direct SPLUNK_ISSUERS ConfigMap verification

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nk runbook

- fix(P2): architecture diagram showed Browser→Splunk→SAIA implying Splunk
  proxies API calls; browser calls SAIA directly with the JWT token Splunk
  issued. Diagram and note now show the correct two-path flow: browser→Splunk
  for auth, browser→SAIA for all AI Assistant API calls. Firewall note added
  (SAIA must be reachable from browser network, not just from Splunk host)
- fix(P3): overview said allowlist is controlled via AIPlatform CR→AIService;
  the reconciler seeds SPLUNK_ISSUERS directly in the ConfigMap. Replaced
  the stale propagation chain with "ConfigMap <name>-saia-config" reference
  matching Step 4
- fix(P1): Step 4 patch examples used <PUBLIC_IP> — SPLUNK_ISSUERS must be
  the exact issuer_uri value (IP or FQDN) from authentication.conf; a
  mismatch causes persistent 401/issuer-not-allowed even after patching.
  Replaced with <EXACT_ISSUER_URI> and added an explanatory note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…doc-v2

docs(cluster_setup): add external Splunk integration runbook
…o to v0.52.0

Addresses VULN-96668 (golang.org/x/net) and VULN-96647 (golang.org/x/crypto).
Also pulls in transitive upgrades: sys v0.45.0, term v0.43.0, text v0.37.0, tools v0.44.0.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… warn on mutable image tags

configure_images() snapshotted artifacts.yaml/splunk-operator-cluster.yaml
to a .original file on first run and restored from it on every subsequent
run before re-injecting image values. Since .original was never refreshed,
any legitimate change to those manifests (new operator release, new env
var, new sidecar) was silently reverted on every re-install after the
first. Drop the snapshot/restore step; the sed substitutions already only
touch their own named fields, so rewriting in place is idempotent.

Also warn in validate_image_config() when a configured image has no tag
or uses a mutable tag (:latest, :preview, :stable*, :dev, :nightly),
since imagePullPolicy: IfNotPresent means re-running install without a
new distinct tag will not actually upgrade the running image.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4b8c88dd52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/cluster_setup/k0s_cluster_with_stack.sh
The image: substitutions for the AI operator and Splunk operator matched
by content (*splunk*ai*operator*, *splunk*operator*) against the image
string itself. This only matched the manifest's pristine default image;
once a custom image (e.g. a private registry with no such substring) was
written in on a prior run, a later run with a new tag would silently fail
to match, leaving the stale image in place. This was previously masked by
a since-removed snapshot/restore step that reset the manifest to pristine
before each run.

Anchor both substitutions on the unique env entry that always immediately
precedes the target image: line (RAY_VERSION for artifacts.yaml, POD_NAME
for splunk-operator-cluster.yaml) instead of matching image content.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kupratyu-splunk

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b07745c4ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .gitignore
Comment thread tools/cluster_setup/k0s_cluster_with_stack.sh Outdated
Comment thread tools/cluster_setup/k0s_cluster_with_stack.sh
kupratyu-splunk and others added 2 commits July 20, 2026 12:13
…rades

fix(deps): upgrade golang.org/x/net to v0.55.0 and golang.org/x/crypt…
Three follow-up issues from PR review on the prior idempotency fix:

- Restore the tools/cluster_setup/*.original .gitignore rule (narrowed
  comment): eks_cluster_with_stack.sh still snapshots/restores .original
  backups, so deleting the rule left those EKS byproducts untracked in
  every checkout even though k0s_cluster_with_stack.sh no longer needs it.

- Parse the tag from the last path segment only in the mutable-tag check,
  not from the whole image string. A registry port (e.g.
  localhost:5000/team/saia-api) was previously read as the tag, so
  untagged images behind a ported registry silently skipped the
  no-tag warning.

- Extend the mutable-tag check to cover images.fluentBit.image and
  images.nginx.image (always patched via RELATED_IMAGE_FLUENT_BIT/
  RELATED_IMAGE_NGINX) and images.splunk.image (patched via
  RELATED_IMAGE_SPLUNK_ENTERPRISE, guarded by SPLUNK_MODE=internal since
  it's only deployed in that mode) — these were patched into the
  manifest but never checked for the same same-tag-rerun hazard.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kupratyu-splunk

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75e75ee8b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/cluster_setup/k0s_cluster_with_stack.sh
kupratyu-splunk and others added 2 commits July 21, 2026 12:44
configure_images() patches RELATED_IMAGE_OTEL_COLLECTOR into the
manifest, but images.otelCollector.image was missing from the
mutable-tag validation list, so an untagged or mutable-tagged OTel
collector image silently skipped the same-tag-rerun warning.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kupratyu-splunk

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c271c4ea1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/cluster_setup/k0s_cluster_with_stack.sh Outdated
kupratyu-splunk and others added 2 commits July 21, 2026 14:31
configure_images() patches images.splunk.operatorImage into
splunk-operator-cluster.yaml (via the POD_NAME-anchored image: sed) in
internal mode, but the mutable-tag validation loop never checked
SPLUNK_OPERATOR_IMAGE — so a tagless or mutable-tagged operator image
was validated without warning.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add two unit-test suites to test_k0s_cluster_with_stack.sh:

- configure_images upgrade idempotency: reproduces the exact regression
  fixed in b07745c — a custom private-registry image containing none of
  "splunk"/"ai"/"operator" must be correctly overwritten on a second
  ("upgrade") run, not left stale.

- validate_image_config mutable-tag warnings: covers the registry-port
  tag-parsing fix and every RELATED_IMAGE_* field configure_images
  patches (operator, ray, weaviate, saia x3, fluentBit, nginx,
  otelCollector, splunk + splunk-operator under internal mode), so any
  future field added to one list without the other gets caught.

Track test_installer_dry_run.sh (previously untracked) and add a new
cluster-setup-tests.yml workflow that runs both bash test suites,
wired into main.yml alongside unit-tests. Neither suite needs a
cluster, kubectl, or network access.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kupratyu-splunk
kupratyu-splunk merged commit 2d7b1a8 into ai-tier-ga Jul 26, 2026
@kupratyu-splunk
kupratyu-splunk deleted the fix/k0s-install-image-upgrade branch July 26, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant